Auto exposure techniques for variable lighting conditions

ABSTRACT

Systems, methods, and a computer readable medium for performing auto exposure (AE) techniques that are beneficial in variable lighting conditions—and particularly applicable to handheld and/or mobile videoconferencing applications—are disclosed herein. Handheld and/or mobile videoconferencing applications—unlike their fixed camera counterparts—are often exposed to a wide variety of rapidly changing lighting and scene conditions, and thus face a difficult trade-off between adjusting exposure parameter values too frequently or not frequently enough. In personal electronic devices executing such handheld and/or mobile videoconferencing applications, it may be desirable to: use a small, centered, and center-weighted exposure metering region; set a relatively low brightness target value; and adjust the camera&#39;s exposure parameter values according to a distance-dependent convergence speed function. The use of such techniques, in conjunction with a relatively large stability region, may also improve the quality of a video encoder&#39;s temporal predictions—and thus video quality—in videoconferencing applications.

CROSS-REFERENCE TO RELATED APPLICATION

This Application claims priority to U.S. Provisional Application Ser.No. 61/351,901, entitled, “Auto Exposure Techniques for VariableLighting Conditions” filed Jun. 6, 2010 and which is incorporated byreference in its entirety herein.

BACKGROUND

Today, many personal electronic devices come equipped with digital videocameras. Often, the cameras in these devices perform many functions,such as: image capture, video capture, and videoconferencing.Videoconferencing with handheld and/or mobile personal electronicdevices is a much more difficult endeavor than traditionalvideoconferencing applications, wherein the camera is mostly fixed andthe lighting and/or composition of the scenery around thevideoconferencing participant(s) does not often change drastically.Thus, it is important that digital video cameras in handheld and/ormobile personal electronic devices be able to capture visually appealingimages in a wide variety of lighting and scene conditions with limitedor no interaction from the user, while at the same time allowing for thevideo encoding of the captured image frames to be carried out in themost computationally effective and visually appealing manner.

One feature that has been implemented in some digital cameras tocompensate for the lack of dynamic range of the relatively small imagesensors typically found in handheld and/or mobile personal electronicdevices is known as “auto exposure.” Auto exposure (AE) can be definedgenerally as any algorithm that automatically calculates and/ormanipulates certain camera exposure parameter values, e.g., exposuretime, gain, or f-number, in such a way that the currently exposed sceneis captured in a desired manner. For example, there may be apredetermined optimum brightness value for a given scene that the camerawill try to achieve by adjusting the camera's exposure value. Exposurevalue (EV) can be defined generally as: log₂N²/t, wherein N is therelative aperture (f-number), and t is the exposure time (i.e., “shutterspeed”) expressed in seconds. Some auto exposure algorithms calculateand/or manipulate the exposure parameters, e.g., the camera's gain, suchthat a mean, center-weighted mean, median, or more complicated weightedvalue (as in matrix-metering) of the image's brightness will equal apredetermined optimum brightness value in the resultant, auto exposedscene.

In particular, AE methods that are tuned for fixed cameravideoconferencing applications typically “meter,” i.e., calculate thebrightness, on a relatively large area of the image, with a reasonablytight “AE stability region,” i.e., range of acceptable brightnessvalues, centered around a fixed, target brightness value (an “AETarget”). In such fixed camera videoconferencing applications, thetarget brightness value may often be set at or above 18% of the maximumcapturable signal strength of the image sensor. The relatively hightarget brightness values and relatively tight stability regionstypically employed in fixed camera videoconferencing applications can beproblematic for handheld and/or mobile videoconferencing applications,in which the scene can be strongly back lit or front lit, and in whichthe scene lighting levels can change often and dramatically.

In mobile videoconferencing applications, it is also paramount to ensurethe highest possible video quality at the lowest possible bit rate.Existing video codec standards today, e.g., H.263 or H.264, employ bothinter-coding and intra-coding techniques. Intra-coding techniques areperformed relative to information that is contained only within thecurrent video frame and not relative to any other frame in the videosequence. Inter-coding techniques, on the other hand, involve temporalprocessing, that is, rather than resending all the information for asubsequent video frame, the codec will only encode and send the changesin pixel location and pixel values from one video frame to the nextvideo frame, while making “temporal predictions” to attempt to takeadvantage of temporal redundancy between subsequent image frames, thusallowing for higher compression rates. This is an effective techniquebecause, often, a large number of the pixels will not change from onevideo frame to the next, and it would be redundant to resend all theimage information with each video frame. However, the more gradually anAE method changes exposure parameter values, the longer the imageframe's average brightness levels are changing, thus making it moredifficult for the encoder to make accurate temporal predictions forinter-coding.

Thus, there is need for systems, methods, and a computer readable mediumfor intelligently and dynamically setting a digital video camera'sexposure parameters in such a way as to lead to visually pleasing imagesand efficiently encoded video streams in handheld and/or mobilevideoconferencing applications.

SUMMARY

Auto exposure (AE) algorithms in handheld and/or mobilevideoconferencing applications—unlike their fixed cameracounterparts—are often exposed to a wide variety of rapidly changinglighting and scene conditions, and thus face a difficult trade-offbetween adjusting exposure parameter values too frequently or notfrequently enough. In personal electronic devices executing suchhandheld and/or mobile videoconferencing applications, it may bedesirable to adjust exposure parameter values gradually, so as to reducevideo oscillations causing visually jarring effects on the device'spreview screen due to rapidly changing brightness levels. However,adjusting the exposure parameter values too gradually can cause thevideo encoder to make inaccurate temporal predictions, resulting in poorquality video streams over a large number of frames. Thus, the inventorshave discovered various techniques to improve auto exposure methods foruse in variable lighting conditions, especially as applied to handheldand/or mobile videoconferencing applications executing on personalelectronic devices.

In one embodiment, a small, centered exposure metering region, e.g., anexposure metering rectangle, is used to meter on the subject of thevideo capture. With handheld videoconferencing applications, the user isgenerally able to keep the camera pointed at the subject of interest,such that the subject of interest is almost always centered in the imageframe. By utilizing a small, centered exposure metering region with acenter-weighted metering weighting matrix, this embodiment takesadvantage of the likely-centered nature of the subject of interest inthe image frame.

In another embodiment, a large “AE stability region” is used to aid inthe AE method's determination of when (and to what degree) to adjustexposure parameter values. The AE stability region defines a range ofacceptable brightness values for the image frame. The AE stabilityregion may be centered on a fixed, target brightness value. If the “AEAverage” for the image frame, i.e., the calculated weighted brightnessaverage over the desired exposure metering region(s), is within the AEstability region, the embodiment may leave the camera's exposureparameter values at their current levels. However, if the “AE Average”for the image frame is outside of the AE stability region (either higherthan the upper limit of the AE stability region or lower than the lowerlimit of the AE stability region), the embodiment may change thecamera's exposure parameter values appropriately to adjust the nextframe's AE Average back towards the stability region by a predeterminedstep size. In one example, the upper limit of the AE stability region ismore than twice the lower limit of the AE stability region, and the AEstability region is centered on the target brightness value for thescene, i.e., the “AE Target.” Such an AE stability region range islarger than what is typically utilized in traditional, fixed cameravideoconferencing applications. Using this larger AE stability regionrange helps to keep the exposure stable in high contrast scenes, such asthose that are strongly back lit or strongly front lit. The larger AEstability region range also helps to reduce oscillations during exposureconvergence, as will be described in further detail below.

In yet another embodiment, a fast “AE convergence speed” is used. AEconvergence speed may be defined as the rate by which the AE methodattempts to bring the scene's AE Average back into an acceptable AEstability region, and AE convergence speed may be manipulated bychanging the magnitude of a step size parameter, as will be discussedfurther below. A fast AE convergence speed, along with a large AEstability region, can improve the appearance of a scene lightingtransition by quickly reaching a new “stable” AE Average point, i.e., anAE Average value within the AE stability region. Using a fast AEconvergence speed may also aid in the efficiency of the video encodingin videoconferencing applications. In preferred embodiments, the AEconvergence speed should be fast, but limited to avoid unwanted videooscillations. As the AE stability region increases in size, the AEconvergence speed can be made faster.

In yet another embodiment, a lower AE Target is used than is typicallyemployed in traditional, fixed camera videoconferencing applications.This helps to reduce over-exposure in embodiments using a small,centered and center-weighted exposure metering region. In one example,the AE Target brightness value is set to no more than 17% of the maximumsignal strength capturable by the image sensor.

Because of efficiencies and favorable results achieved by theembodiments disclosed herein, the auto exposure techniques for variablelighting conditions described below may be implemented directly in apersonal electronic device's hardware and/or software, making thetechniques readily applicable to any number of handheld and/or mobilepersonal electronic devices possessing digital video cameras, e.g.,mobile phones, personal data assistants (PDAs), portable music players,or laptop/tablet computers. Alternatively, the auto exposure techniquesfor variable lighting conditions described below may also be implementedin conventional cameras or traditional, fixed camera videoconferencingapplications.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a typical outdoor scene with a human subject, inaccordance with one embodiment.

FIG. 2 illustrates a typical outdoor scene with a human subjectutilizing a personal electronic device executing a handheld and/ormobile videoconferencing application, in accordance with one embodiment.

FIG. 3 illustrates a typical outdoor scene with a human subject asviewed on a personal electronic device's preview screen, in accordancewith one embodiment.

FIG. 4 illustrates a typical exposure metering region for an outdoorscene with a human subject, in accordance with one embodiment.

FIG. 5 illustrates a center-weighted metering weighting matrix for atypical exposure metering region over an outdoor scene with a humansubject, in accordance with one embodiment.

FIG. 6 illustrates a handheld and/or mobile videoconferencingapplication executed on a personal electronic device's preview screen,in accordance with one embodiment.

FIG. 7 illustrates a graph representing the AE Averages of twohypothetical scenes over time and an AE stability region, in accordancewith one embodiment.

FIG. 8 illustrates, in flowchart form, one embodiment of a process forauto exposure in a handheld/mobile personal electronic device using anAE stability region.

FIG. 9 illustrates, in flowchart form, one embodiment of a process fordetermining AE convergence speeds.

FIG. 10 illustrates a simplified functional block diagram of a personalelectronic device, in accordance with one embodiment.

DETAILED DESCRIPTION

This disclosure pertains to systems, methods and a computer readablemedium for intelligently and dynamically setting a digital videocamera's exposure parameters in such a way as to lead to visuallypleasing images and efficiently encoded video streams in handheld and/ormobile videoconferencing applications, especially in scenes withvariable lighting conditions. While this disclosure discusses newtechniques for AE in handheld and/or mobile videoconferencingapplications, one of ordinary skill in the art would recognize that thetechniques disclosed may also be applied to other contexts andapplications as well. The techniques disclosed herein are applicable toany number of electronic devices with digital image sensors, such asdigital cameras, digital video cameras, mobile phones, personal dataassistants (PDAs), portable music players, computers, and conventionalcameras. An embedded processor, such a Cortex® A8 with the ARM® v7-Aarchitecture, provides a versatile and robust programmable controldevice that may be utilized for carrying out the disclosed techniques.(CORTEX® and ARM® are registered trademarks of the ARM Limited Companyof the United Kingdom.)

In the interest of clarity, not all features of an actual implementationare described in this specification. It will of course be appreciatedthat in the development of any such actual implementation (as in anydevelopment project), numerous decisions must be made to achieve thedevelopers' specific goals (e.g., compliance with system- andbusiness-related constraints), and that these goals will vary from oneimplementation to another. It will be appreciated that such developmenteffort might be complex and time-consuming, but would nevertheless be aroutine undertaking for those of ordinary skill having the benefit ofthis disclosure. Moreover, the language used in this disclosure has beenprincipally selected for readability and instructional purposes, and maynot have been selected to delineate or circumscribe the inventivesubject matter, resort to the claims being necessary to determine suchinventive subject matter. Reference in the specification to “oneembodiment” or to “an embodiment” means that a particular feature,structure, or characteristic described in connection with theembodiments is included in at least one embodiment of the invention, andmultiple references to “one embodiment” or “an embodiment” should not beunderstood as necessarily all referring to the same embodiment.

Referring now to FIG. 1, a typical outdoor scene 100 with a humansubject 102 is shown, in accordance with one embodiment. The scene 100also includes the Sun 106 and a natural object, tree 104. Regions aroundthe face of human subject 102 will likely have much lower luminancevalues than regions around objects such as the Sun 106. This isimportant because, especially in the case of outdoor scenes, such asthat shown in FIG. 1, the Sun 106 (or any number of other possible backlighting sources) can have a large—and often detrimental—effect on theway a camera using a standard exposure algorithm meters the scene.Because of the very large brightness values that will be measured inpixels in the upper half of the scene due to the Sun, cameras using astandard exposure algorithm will tend to meter and expose the scene insuch a manner that the person's face will be quite dark and thebackground will be more fully exposed.

Referring now to FIG. 2, a typical outdoor scene 100 with a humansubject 102 utilizing a personal electronic device 208 executing ahandheld and/or mobile videoconferencing application is shown, inaccordance with one embodiment. The dashed lines 212 indicate theviewing angle of the camera (not shown in FIG. 2) on the reverse side ofcamera device 208, i.e., the side of camera device 208 facing humansubject 102 in FIG. 2. As mentioned previously, although camera device208 is shown here as a mobile phone, the teachings presented herein areequally applicable to any electronic device possessing a camera, suchas, but not limited to: digital video cameras, personal data assistants(PDAs), portable music players, laptop/desktop/tablet computers, orconventional cameras. As illustrated in FIG. 2, the camera device 208may also have a camera 214 on the side of the device that is not facingthe human subject 102 during the use of the mobile videoconferencingapplication. Although the description herein will focus primarily on thecamera (not shown in FIG. 2) on the side of camera device 208 facinghuman subject 102 during the use of the mobile videoconferencingapplication, it will be understood that either such camera may beutilized in conjunction with the improved AE techniques disclosedherein.

Referring now to FIG. 3, a typical outdoor scene 100 with a humansubject 102 as viewed on a camera device 208's preview screen 210 isshown, in accordance with one embodiment. As shown in FIG. 3, the scenedisplayed on camera device 208's preview screen 210 has been captured byfront-facing camera 300 (recall FIG. 2, which shows human subject 102holding camera device 208 at roughly arm's length and pointed at hisface). As mentioned above, with handheld videoconferencing applications,the user is generally able to keep the camera pointed at the subject ofinterest (oftentimes himself or herself), such that the subject ofinterest is almost always centered in the image frame of preview screen210. As shown in FIG. 3, the human subject of interest 102 is nearly,though not perfectly, centered on preview screen 210.

Referring now to FIG. 4, a typical exposure metering region 400 for anoutdoor scene 100 with a human subject 102 is shown, in accordance withone embodiment. In this exemplary embodiment, the exposure meteringregion is represented by exposure metering rectangle 400 and hasdimensions that are one third of the corresponding dimensions of cameradevice 208's preview screen 210. That is, exposure metering rectangle400's width 406 is one third of the width of camera device 208's previewscreen 210 and exposure metering rectangle 400's height 402 is one thirdof the height of camera device 208's preview screen 210. The one thirddimension choice is not strictly necessary, but it has been empiricallydetermined that choosing an exposure metering rectangle 400 of this sizecan help exposure determinations from being overly influenced by lightsources located at the periphery of the frame, e.g., overhead lights atthe top of an image. In another embodiment the dimensions of theexposure metering rectangle are no more than 35% of the correspondingdimensions of the camera device's preview screen. In yet anotherembodiment, it has been empirically determined that the longer dimension402 of the exposure metering rectangle 400 may be 50% of the size of thelonger dimension of the camera device 208's preview screen 210, whilethe shorter dimension 406 of the exposure metering rectangle is set tobe 35% of the size of the shorter dimension of the camera device 208'spreview screen 210.

In another embodiment, the dimensions of the exposure metering rectangleare not fixed to a predetermined portion of the dimensions of the cameradevice's preview screen. Rather, the dimensions of the exposure meteringrectangle may be chosen such that they subtend a particular maximumcentral angle coming outwards from the exposure metering rectangle ofthe camera. In one example, the width and height of the exposuremetering rectangle each subtend a maximum central angle of 35 degrees.In another example, the longer dimension of the exposure meteringrectangle subtends a maximum central angle of 50 degrees, whereas theshorter dimension of the exposure metering rectangle subtends a maximumcentral angle of 35 degrees. Determining the dimensions of the exposuremetering region based on a maximum subtended central angle rather thanon a fixed portion of the camera device's preview screen may beadvantageous in certain situations, such as when a person or object ofinterest is very close or far from the camera.

In auto exposure algorithms according to some embodiments disclosedherein, an exposure metering region, such as exposure metering rectangle400, remains centered in the frame, and the camera's exposure parametersare driven such that the weighted average brightness of the pixelswithin exposure metering rectangle 400 are equal or nearly equal to anAE Target brightness value, e.g., a 17% gray value. For example, with animage sensor capable of recording 10-bit luminance (i.e., brightness)values, the maximum luminance value is 2¹⁰, or 1024, and, thus, a 17%gray value would be 1024*0.17, or approximately 174. If the scene issufficiently brighter than the “target” 17% gray value, the cameracould, e.g., decrease the camera's gain, whereas, if the scene weresufficiently darker than the optimum 17% gray value, the camera could,e.g., increase the camera's gain. A small, centered exposure meteringregion, such as that explained above with regard to FIG. 3 may worksatisfactorily for a scene that has a human subject centered in it, butmay need to be adjusted if there is not a human subject in the scene, orif the human subject is not centered in the scene. Such adjustments tothe exposure metering region could occur either manually orautomatically.

Referring now to FIG. 5, a center-weighted metering weighting matrix 500for a typical exposure metering region 400 over an outdoor scene 100with a human subject 102 is shown, in accordance with one embodiment. Asshown in FIG. 5, center-weighted metering weighting matrix 500 consistsof 64 equally-sized regions—eight rows and eight columns—that have beenassigned various “weights” comprising metering weighting matrix 500,represented by the numbers located in each of the 64 regions withinexposure metering region 400. In one embodiment, the weights for theregions may range from 0-15, although a greater or smaller range ofpossible weights (as well as a larger or smaller number of regions) maybe employed if such a scheme is empirically deemed to be beneficial. Themetering weighting matrix 500 shown in FIG. 5 is said to be“center-weighted” because the weighted values increase radially from theouter edges towards the center of the metering weighting matrix, as seenby the values ranging from 1 around the outer edge of exposure meteringregion 400 to 9 near the center of exposure metering region 400.

By summing the products of each region's average luminance (i.e., theaverage luminance value for the pixels comprising the region) and theregion's “weight” value from the metering weighting matrix, and thendividing that summed value by the sum of the weights of all the regionsin the metering weighting matrix, a weighted average value, referred toherein as the “AE Average” or a “weighting matrix adjusted mean” may becalculated. This is merely one formula for calculating a weightedaverage of the scene's luminance values, and other formulas may beempirically determined and applied in order to calculate the “AEAverage” in a desired manner. Based on the value of the AE Average for agiven scene, the auto exposure method may be able to drive the camera'sexposure parameter values in an intelligent and desirable way. Theweighting scheme and values described above are merely exemplary and areonly one embodiment of a scheme that has been empirically determined toproduce satisfactory images for most scene compositions. Further,different weighting schemes may be appropriate for differentapplications. For example, different weighting schemes may be employedfor taking a photo, taking a video, or videoconferencing.

Referring now to FIG. 6, a handheld and/or mobile videoconferencingapplication being executed on personal electronic device 208's previewscreen 210 is shown, in accordance with one embodiment. In thisembodiment, human subject 102, located in scene 100 is engaged in avideoconference with a far-end video conferencing unit currently pointedat indoor scene 600 with human subject 602. In order to keep the camera300 pointed at his face, human subject 102 may be presented with a smallpreview window 604 at some location on preview screen 210 indicating theimage frame that is currently being transmitted to far-end videoconferencing participant 602's videoconferencing device. By utilizingpreview window 604, human subject 102 may ensure that his face remainscentered in the image frame, such that a small, centered andcenter-weighted exposure metering region (such as that described inrelation to FIG. 5) will continue to produce satisfactory auto exposedimage frames. The location and size of preview window 604 may becustomizable by the user of camera device 208 or may optionally betoggled off.

Referring now to FIG. 7, a graph 700 with two lines 704/706representative of the AE Average values over time of two hypotheticalscenes and an AE stability region 702 are shown, in accordance with oneembodiment. The y-axis of graph 700 represents the AE Average for agiven image frame. The x-axis of graph 700 represents time, as measuredby the number of frames captured. Each vertical “tick mark” on thex-axis is meant to represent a subsequent frame captured by the imagesensor. In one embodiment, a relatively large AE stability region 702(as compared to the AE stability region typically used in traditional,fixed camera videoconferencing applications) is used to aid in the AEmethod's determination of when (and to what degree) to adjust thecamera's exposure parameter values. The AE stability region 702 definesa range of acceptable AE Average brightness values for the image frame.The AE stability region 702 may be centered on a fixed, AE Target 710brightness value, e.g., a 17% gray value. If the “AE Average” for theimage frame, i.e., the calculated weighted brightness average over thedesired exposure metering region(s), is within the AE stability region702, the embodiment may leave the camera's exposure parameter values attheir current levels. However, if the “AE Average” for the image frameis outside of the AE stability region 702 (either higher than the upperlimit of the AE stability region 708 or lower than the lower limit ofthe AE stability region 712), the embodiment may change the camera'sexposure parameter values appropriately to attempt to adjust the nextframe's AE Average back towards the stability region 702. In oneexample, the upper limit of the AE stability region 708 is more thantwice the lower limit of the AE stability region 712, and the AEstability region 702 is centered on the AE Target 710. One set ofexemplary values for the AE stability region 702 is: a 24% gray valuefor the upper limit 708, a 17% gray value for the AE Target 710, and a10% gray value for the lower limit 712 of the AE stability region. Suchan AE stability region 702 range is larger than what is typicallyutilized in traditional, fixed camera videoconferencing applications.Using this larger AE stability region 702 range helps to keep theexposure stable in high contrast scenes, such as those that are stronglyback lit or strongly front lit. The larger AE stability region 702 rangealso helps to reduce oscillations during auto exposure convergence, aswill be described in further detail below.

AE convergence is the process by which an AE method attempts to adjustthe AE Average of the scene being captured back to an acceptablebrightness range, e.g., within an AE stability region. The rate at whichthe AE method attempts to adjust the AE Average of the scene may bereferred to herein as the “AE convergence speed.” In some embodimentsherein, a relatively fast AE convergence speed (as compared to thosetypically used in traditional, fixed camera videoconferencingapplications) is used. A fast AE convergence speed, along with arelatively large AE stability region, can improve the appearance of ascene lighting transition by quickly reaching a new stable AE Averagepoint. Using a fast AE convergence speed may also aid in the efficiencyof the video encoding in videoconferencing applications. In preferredembodiments, the AE convergence speed should be fast, but limited toavoid unwanted video oscillations. As the AE stability region 702increases in size, the AE convergence speed can be made faster.

In one embodiment, in an attempt to change the scene's AE Average, theAE method may adjust the camera's exposure parameter values for eachframe that is captured by the camera's image sensor. For each frame thatis captured, a “brightness change step size” may first be calculated.The “brightness change step size” will define the magnitude of change inbrightness that the AE method will attempt to impose on the next imageframe captured by the camera's image sensor. The larger the magnitude ofthe step size, the fast the AE convergence speed will be. In oneembodiment, the “brightness change step size” is calculated according tothe following formula:Brightness_Change_Step_Size=alpha*|(Current_AE_Average−AE_Target)|;where: 0.2<alpha<1 (Equation 1). In a preferred embodiment, alpha is setto be no smaller than the fractional value: 60/255.

By calculating a brightness change step size that is dependent on thedistance, i.e., the magnitude of the difference, between the current AEAverage and the AE Target, the AE method is able to adjust the camera'sexposure parameter values to attempt to quickly bring the AE Averageback within the AE stability region. However, if the AE stability regionis not sufficiently large, the use of too large of a convergence stepsize could have the unwanted effect of causing the AE Average toactually “jump over” the AE stability region from one frame to the nextor over the course of several frames. For example, if the AE Average ismuch higher than upper limit of the AE stability region, Equation 1above will calculate a large step size which, if the AE stability regionis sufficiently small, could cause the AE Average to actually be belowthe lower limit of the AE stability region in the next frame. This typeof constant video oscillation can cause unwanted flickering on thecamera device's preview screen and lead to difficulties in temporalpredictions for the video encoder. A more preferred embodiment wouldemploy a large AE stability range, as well as a fast AE convergencespeed (i.e., a large step size), but stop adjusting the camera'sexposure parameter values as soon as the AE Average for the scenereturned to the accepted AE stability region. It has been empiricallydetermined that a smaller number of “bad” image frames in avideoconferencing application, e.g., during a rapid lighting changes, ismore visually appealing to users than a longer period of “bad” imageframes as the AE method gradually changes the camera's exposureparameter values over time to bring the AE Average for the scene backinto an acceptable AE stability region. Thus, a fast, distance-dependentAE convergence speed—combined with a large AE stability region—arepreferable techniques for handheld and/or mobile videoconferencingapplications.

By way of example, line 704 in FIG. 7 represents the AE Average overtime of a scene being captured by a handheld and/or mobile personalelectronic device executing a videoconferencing application. As shown inFIG. 7, the scene represented by line 704 always remained within the AEstability region 702. Thus, the AE method never needed to adjust thecamera's exposure parameter values to attempt to bring the AE Averageback within the acceptable AE stability region. By contrast, line 706 inFIG. 7 represents a scene whose AE Average began within the AE stabilityregion 702 but that, at captured frame 20 (labeled f20 in FIG. 7),exceeded the upper limit 708 of the AE stability region 702. At thatpoint, the AE method operating on the scene represented by line 706acted to change the camera's exposure parameter values to attempt toadjust the next frame's AE Average back towards the AE stability region702 by an amount equal to the calculated “brightness change step size.”As shown in FIG. 7, by the fifth captured frame after the AE Averageleft the AE stability region (labeled f25 in FIG. 7), the AE method wasable to cause the scene's brightness, i.e., AE Average, to converge backto the AE stability region. At that point, the scene represented by line706 maintained an AE Average within the AE stability region, so nofurther action was needed from the AE method.

Referring now to FIG. 8, one embodiment of a process for auto exposurein a handheld/mobile personal electronic device using an AE stabilityregion is shown in flowchart form. First, the process begins at Step800. Next, the user points the camera at the desired scene (Step 802).Next, the AE method creates an exposure metering region(s) over thedesired areas over the scene (Step 804). At this point, the AE methodmay begin to process image frames being captured by the image sensor(Step 806). A weighted AE Average is calculated for the current frameusing the desired exposure metering region(s) and one or morepredetermined metering weighting matrixes (Step 808). Once the AEAverage is calculated for the current frame, the AE method may determinewither the AE Average is within the AE stability region (Step 810). Ifthe AE Average is within the AE stability region, the process mayproceed to Step 814 and leave the camera's exposure parameters at theircurrent values. If instead, the AE Average is not within the AEstability region, the process may proceed to FIG. 9 (Step 812), wherebrightness changes step size may be calculated and appropriateadjustments may be made to the camera's exposure parameter values, as isexplained in further detail below in relation to FIG. 9. At this point,the current frame's image data may be sent to a video encoder (Step 816)and, once encoded, be sent to a far-end videoconferencing unit to bereceived, decoded, and displayed (Step 818). At Step 816, once thecurrent image frame has been captured, the process simply returns toStep 806 to capture the next image frame, calculate its AE Average, makethe appropriate adjustments to the camera's exposure parameter values ifneeded, and repeat the process of FIG. 8 again until the user indicatesa desire to no longer execute the image capture, video capture, orvideoconferencing application.

Referring now to FIG. 9, one embodiment of a process for determining AEconvergence speeds is shown in flowchart form. First, the process beginsat Step 900. Next, brightness change step size may be calculatedaccording to Equation 1 above (Step 902). Next, the AE method changesthe camera's exposure parameter values to adjust the next frame's AEAverage back towards the AE stability region by an amount equal to thecalculated brightness change step size (Step 904). At this point, theprocess of FIG. 9 ends (Step 906), and control of the AE method may bereturned to Step 816 in FIG. 8.

Referring now to FIG. 10, a simplified functional block diagram of arepresentative personal electronic device 1000 according to anillustrative embodiment, e.g., a mobile phone possessing one or morecamera devices, such as camera device 208, is shown. The personalelectronic device 1000 may include a processor 1016, storage device1014, user interface 1018, display 1020, coder/decoder (CODEC) 1002, bus1022, memory 1012, communications circuitry 1010, a speaker ortransducer 1004, a microphone 1006, and one or more image sensors withassociated camera hardware 1008. Processor 1016 may be any suitableprogrammable control device and may control the operation of manyfunctions, such as the auto exposure algorithm discussed above, as wellas other functions performed by personal electronic device 1000.Processor 1016 may drive display 1020 and may receive user inputs fromthe user interface 1018.

Storage device 1014 may store media (e.g., photo and video files),software (e.g., for implementing various functions on device 1000),preference information (e.g., media playback preferences), personalinformation, and any other suitable data. Storage device 1014 mayinclude one more storage mediums, including for example, a hard-drive,permanent memory such as ROM, semi-permanent memory such as RAM, orcache.

Memory 1012 may include one or more different types of memory which maybe used for performing device functions. For example, memory 1012 mayinclude cache, ROM, and/or RAM. Bus 1022 may provide a data transferpath for transferring data to, from, or between at least storage device1014, memory 1012, and processor 1016. CODEC 1002 may be included toconvert digital audio signals into analog signals for driving thespeaker 1004 to produce sound including voice, music, and other likeaudio. The CODEC 1002 may also convert audio inputs from the microphone1006 into digital audio signals for storage in memory 1012 or storage1014. The CODEC 1002 may include a video CODEC for processing digitaland/or analog video signals.

User interface 1018 may allow a user to interact with the personalelectronic device 1000. For example, the user input device 1018 can takea variety of forms, such as a button, keypad, dial, a click wheel, or atouch screen. Communications circuitry 1010 may include circuitry forwireless communication (e.g., short-range and/or long rangecommunication). For example, the wireless communication circuitry may beWi-Fi®) enabling circuitry that permits wireless communication accordingto one of the 802.11 standards. (Wi-Fi® is a registered trademark of theWi-Fi Alliance.) Other wireless network protocols standards could alsobe used, either as an alternative to the identified protocols or inaddition to the identified protocols. Other network standards mayinclude BLUETOOTH®, the Global System for Mobile Communications (GSM®),and code division multiple access (CDMA) based wireless protocols.(BLUETOOTH® is a registered trademark of Bluetooth SIG, Inc., and GSM®is a registered trademark of GSM Association.) Communications circuitry1010 may also include circuitry that enables device 1000 to beelectrically coupled to another device (e.g., a computer or an accessorydevice) and communicate with that other device.

In one embodiment, the personal electronic device 1000 may be a personalelectronic device dedicated to processing media such as audio and video.For example, the personal electronic device 1000 may be a media devicesuch as media player, e.g., an MP3 player, a game player, a remotecontroller, a portable communication device, a remote orderinginterface, an audio tour player, or other suitable personal device. Thepersonal electronic device 1000 may be battery-operated and highlyportable so as to allow a user to listen to music, play games or video,record video or take pictures, communicate with others, and/or controlother devices. In addition, the personal electronic device 1000 may besized such that it fits relatively easily into a pocket or hand of theuser. By being handheld, the personal computing device 1000 may berelatively small and easily handled and utilized by its user and thusmay be taken practically anywhere the user travels.

As discussed previously, the relatively small form factor of certaintypes of personal electronic devices 1000, e.g., personal media devices,enables a user to easily manipulate the device's position, orientation,and movement. Accordingly, the personal electronic device 1000 mayprovide for improved techniques of sensing such changes in position,orientation, and movement to enable a user to interface with or controlthe device 1000 by affecting such changes. Further, the device 1000 mayinclude a vibration source, under the control of processor 1016, forexample, to facilitate sending motion, vibration, and/or movementinformation to a user related to an operation of the device 1000. Thepersonal electronic device 1000 may also include one or more imagesensors and associated camera hardware 1008, e.g., a front-facing orrear-facing camera, that enables the device 1000 to capture an image orseries of images, i.e., video, continuously, periodically, at selecttimes, and/or under select conditions.

The foregoing description is not intended to limit or restrict the scopeor applicability of the inventive concepts conceived of by theApplicants. As one example, although the present disclosure focused onauto exposure techniques for mobile and/or handheld videoconferencingapplications; it will be appreciated that the teachings of the presentdisclosure can be applied to other contexts, e.g.: traditional fixedcamera videoconferencing applications, conventional cameras, imagecapture, and video capture. In exchange for disclosing the inventiveconcepts contained herein, the Applicants desire all patent rightsafforded by the appended claims. Therefore, it is intended that theappended claims include all modifications and alterations to the fullextent that they come within the scope of the following claims or theequivalents thereof.

1. An auto exposure system comprising: an image sensor for capturing animage representative of a scene; a memory coupled to the image sensor;and a programmable control device communicatively coupled to the imagesensor and the memory, wherein the memory includes instructions forcausing the programmable control device to perform an auto exposuremethod on image information received from the image sensor, the methodcomprising: defining an exposure metering region over a desired area ofthe image; calculating a weighted average value for the exposuremetering region; comparing the weighted average value to a predeterminedrange of acceptable values; and adjusting an exposure parameter for theimage sensor based at least in part on the magnitude of the differencebetween the weighted average value and a center of the predeterminedrange of acceptable values.
 2. The auto exposure system of claim 1,wherein the exposure metering region comprises a rectangular region. 3.The auto exposure system of claim 2, wherein the width of the exposuremetering region subtends a central angle of no more than 35 degrees, andwherein the height of the exposure metering region subtends a centralangle of no more than 35 degrees.
 4. The auto exposure system of claim2, wherein the longer dimension of the exposure metering region subtendsa central angle of no more than 50 degrees, and wherein the shorterdimension of the exposure metering region subtends a central angle of nomore than 35 degrees.
 5. The auto exposure system of claim 1, whereinthe exposure metering region is configured to remain centered in theimage.
 6. The auto exposure system of claim 1, wherein the programmedact of adjusting an exposure parameter for the image sensor based atleast in part on the magnitude of the difference between the weightedaverage value and the center of the predetermined range of acceptablevalues comprises adjusting one or more of the following: an exposuretime, a shutter speed, ISO, a gain level, and an f-number.
 7. The autoexposure system of claim 1, wherein the weighted average value and therange of acceptable values comprise pixel luminance values.
 8. The autoexposure system of claim 1, wherein the programmed act of calculating aweighted average value for the exposure metering region comprisescalculating one or more of the following: a mean, a center-weightedmean, a median, and a weighting matrix adjusted mean.
 9. The autoexposure system of claim 1, wherein the programmed act of adjusting anexposure parameter for the image sensor is configured to occur accordingto a predetermined distance-dependent convergence step size function.10. The auto exposure system of claim 1, wherein the predetermined rangeof acceptable values comprises a lower limit and an upper limit, andwherein the upper limit is at least two times as large as the lowerlimit.
 11. The auto exposure system of claim 10, wherein thepredetermined range of acceptable values is centered on a predeterminedtarget value.
 12. An auto exposure system comprising: an image sensorfor capturing an image representative of a scene; a memory coupled tothe image sensor; and a programmable control device communicativelycoupled to the image sensor and the memory, wherein the memory includesinstructions for causing the programmable control device to perform anauto exposure method on image information received from the imagesensor, the method comprising: defining an exposure metering region overa desired area of the image; calculating a weighted average value forthe exposure metering region; calculating a magnitude of the differencebetween the calculated weighted average value and a predetermined targetvalue; multiplying the calculated magnitude of the difference betweenthe weighted average value and the predetermined target value by apredetermined step size factor, wherein the result of the multiplicationcomprises a first step size value; and adjusting an exposure parameterfor the image sensor based at least in part on the magnitude of thefirst step size value.
 13. The auto exposure system of claim 12, whereinthe programmed act of calculating a weighted average value for theexposure metering region comprises calculating one or more of thefollowing: a mean, a center-weighted mean, a median, and a weightingmatrix adjusted mean.
 14. The auto exposure system of claim 12, whereinthe programmed act of adjusting an exposure parameter for the imagesensor is configured to occur according to a predetermineddistance-dependent convergence step size function.
 15. The auto exposuresystem of claim 12, wherein the predetermined target value is no greaterthan 17% of the maximum signal strength capturable by the image sensor.16. The auto exposure system of claim 12, wherein the predetermined stepsize factor is no less than the fractional value: 60/255.
 17. A mobilevideoconferencing method comprising: receiving image informationrepresentative of the physical scene and comprising a plurality ofpixels from an image sensor at a near-end videoconferencing unit;defining an exposure metering region that is centered over the imageinformation; adjusting an exposure parameter for the image sensor basedat least in part on a comparison of luminance values of pixels in theexposure metering region to a predetermined target luminance value;encoding the received image information into a video stream; and sendingthe encoded video stream to a far-end videoconferencing unit.
 18. Themethod of claim 17, wherein the magnitude of the adjustment to theexposure parameter is based at least in part on the magnitude of thedifference between a weighted average luminance value of pixels in theexposure metering region and the predetermined target luminance value.19. The method of claim 17, wherein the image sensor at the near-endvideoconferencing unit comprises a front-facing camera.
 20. The methodof claim 17, wherein the near-end videoconferencing unit comprises oneor more of the following: a digital camera, a digital video camera, amobile phone, a personal data assistant, a portable music player, acomputer, and a conventional camera.
 21. The method of claim 17, whereinthe weighted average luminance value of pixels in the exposure meteringregion comprises one of the following: a center-weighted mean and aweighting matrix adjusted mean.
 22. The method of claim 17, wherein thepredetermined target luminance value is no greater than 17% of themaximum signal strength capturable by the image sensor.
 23. The methodof claim 17, wherein the width of the exposure metering region subtendsa central angle of no more than 35 degrees, and wherein the height ofthe exposure metering region subtends a central angle of no more than 35degrees.
 24. A method of auto exposing a physical scene comprising:receiving a first image frame from an image sensor, wherein the firstimage frame comprises information representative of the physical sceneand comprising a plurality of pixels; defining an exposure meteringregion that is centered over the first image frame; adjusting anexposure parameter for the image sensor based at least in part on acalculated step size value, wherein the calculated step size value isbased at least in part on a comparison of a weighted average of pixelluminance values in the exposure metering region centered over the firstimage frame to a predetermined target luminance value; receiving asecond image frame from an image sensor, wherein the second image framecomprises information representative of the physical scene andcomprising a plurality of pixels; and defining an exposure meteringregion that is centered over the second image frame, wherein the secondimage frame is representative of the physical scene at a later point intime than the first image frame, and wherein the magnitude of theadjustment to the exposure parameter is configured such that a weightedaverage of pixel luminance values in the exposure metering region of thesecond image frame differs from the weighted average of pixel luminancevalues in the exposure metering region of the first image frame by thestep size value.
 25. A non-transitory computer usable medium having acomputer readable program code embodied therein, wherein the computerreadable program code is adapted to be executed to implement the methodperformed by the programmable control device of claim 1.